Migrate compiletest to use `ui_test`-style `//@` directives #121370

jieyouxu · 2024-02-20T23:56:18Z

Preface

There's an on-going effort to rewrite parts of or the entirety of compiletest
(rust-lang/compiler-team#536). A step towards this involve migrating
compiletest tests to use the ui_test framework, which
involves changing compiletest directives in // <directive-name> style to ui_test
//@ <directive-name> style (rust-lang/compiler-team#512).

This PR aims to implement the directive-style change from // to //@ for the remaining
non-"ui" test suite tests.

Key Changes

All tests/ tests now use //@ directives.
Compiletest only accepts //@ and issues an error if an old-style directive is detected.
// ignore-tidy and // ignore-tidy-* are considered tidy directives and are ignored by
compiletest header parsing.

Diff Generation

The diff is generated by:

Collecting directives from tests/ via hijacking compiletest to emit successfully parsed
directive lines.
Using a migration tool
(https://github.com/jieyouxu/compiletest-ui_test-header-migration/tree/master) to replace
// directives in compiletest tests with //@.

Reproduction Steps

Delete the temporary file $RUSTC_REPO_PATH/build/<target_triple>/test/__directive_lines.txt,
if the collection script was previously ran.

Use the https://github.com/jieyouxu/rust/tree/collect-test-directives collect-test-directives
script, which outputs a temporary file recording headers occuring in each compiletest test.

You need to checkout this branch: git checkout collect-test-directives.
This needs to be rebased on latest master to ensure up-to-date test directives can be collected.

You need to run ./x test on each of the test/* subfolders once:

./x test tests/assembly/ --stage 1 --force-rerun
./x test tests/codegen/ --stage 1 --force-rerun
./x test tests/codegen-units/ --stage 1 --force-rerun
./x test tests/coverage/ --stage 1 --force-rerun
./x test tests/coverage-run-rustdoc/ --stage 1 --force-rerun
./x test tests/debuginfo/ --stage 1 --force-rerun
./x test tests/incremental/ --stage 1 --force-rerun
./x test tests/mir-opt/ --stage 1 --force-rerun
./x test tests/pretty/ --stage 1 --force-rerun
./x test tests/run-make/ --stage 1 --force-rerun
./x test tests/run-make-fulldeps/ --stage 1 --force-rerun
./x test tests/run-pass-valgrind/ --stage 1 --force-rerun
./x test tests/rustdoc/ --stage 1 --force-rerun
TARGET=<target-triple> ./x test tests/rustdoc-gui/ --stage 1 --force-rerun
./x test tests/rustdoc-js/ --stage 1 --force-rerun
./x test tests/rustdoc-js-std/ --stage 1 --force-rerun
./x test tests/rustdoc-json/ --stage 1 --force-rerun
./x test tests/rustdoc-ui/ --stage 1 --force-rerun
./x test tests/ui/ --stage 1 --force-rerun
./x test tests/ui-fulldeps/ --stage 1 --force-rerun

Checkout the migrate-compiletest-directives branch.
Run the migration tool https://github.com/jieyouxu/compiletest-ui_test-header-migration.
Check that the migration at least does not cause test failures if you change compiletest to
accept //@ directives only. This is also required if the test outputs somehow need to be
blessed.
- RUSTC_TEST_FAIL_FAST=1 ./x test tests/<secondary-directory> --stage 1 --bless
Confirm that there is no difference after running the migration tool when you are on the
migrate-compiletest-directives branch.

Follow Up Work

Adjust rustc-dev-guide docs for compiletest directives (this time for all the other suites and modes). Update compiletest directives to be in ui_test style //@ rustc-dev-guide#1895.

rustbot · 2024-02-21T00:09:38Z

Some changes occurred in GUI tests.

cc @GuillaumeGomez

Zalathar · 2024-02-21T01:32:23Z

FYI, the coverage tests will need to have their .coverage snapshots re-blessed (or edited) as well.

Also watch out, because the part of a coverage test that uses those snapshots doesn't run in PR CI jobs, and by default it doesn't run in local builds either.

(See https://rust-lang.zulipchat.com/#narrow/stream/326414-t-infra.2Fbootstrap/topic/Building.20the.20profiler.20runtime.20in.20more.20profiles.2Fjobs.3F for more context on why.)

So you will probably need to set [build] profiler = true in your cargo.toml, to avoid unpleasant surprises when the full CI suite fails.

jieyouxu · 2024-02-21T01:35:35Z

FYI, the coverage tests will need to have their .coverage snapshots re-blessed (or edited) as well.

Also watch out, because the part of a coverage test that uses those snapshots doesn't run in PR CI jobs, and by default it doesn't run in local builds either.

(See https://rust-lang.zulipchat.com/#narrow/stream/326414-t-infra.2Fbootstrap/topic/Building.20the.20profiler.20runtime.20in.20more.20profiles.2Fjobs.3F for more context on why.)

So you will probably need to set [build] profiler = true in your cargo.toml, to avoid unpleasant surprises when the full CI suite fails.

Thanks for the heads up. I'm also expecting other failures too that'll only get picked up in full build lol (just like in #120881).

Since the coverage reports seem to have a reasonably predictable format, I think I might be able to hack up some external script to convert the contained directives.

Zalathar · 2024-02-21T02:02:46Z

Since the coverage reports seem to have a reasonably predictable format, I think I might be able to hack up some external script to convert the contained directives.

Yeah, for the changes you're making this approach should work pretty well.

bjorn3 · 2024-02-21T13:25:50Z

src/tools/compiletest/src/header.rs

 /// names. This is **not** an exhaustive list of all possible directives. Instead, this is a
 /// best-effort approximation for diagnostics.


Is it still the case that this is not an exhaustive list? If not can you remove the respective part of this comment.

It's as exhaustive as the currently used directives go. It is still possible that some combination e.g. ignore-platform-xxx has never been used else where in the test suite.

bors · 2024-02-22T16:39:12Z

⌛ Testing commit cd25009 with merge f62f490...

bors · 2024-02-22T18:39:10Z

☀️ Test successful - checks-actions
Approved by: oli-obk
Pushing f62f490 to master...

rust-timer · 2024-02-22T22:12:18Z

Finished benchmarking commit (f62f490): comparison URL.

Overall result: no relevant changes - no action needed

@rustbot label: -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	2.2%	[2.0%, 2.4%]	2
Improvements ✅ (primary)	-2.8%	[-3.3%, -2.3%]	2
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-2.8%	[-3.3%, -2.3%]	2

Cycles

This benchmark run did not return any relevant results for this metric.

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 649.98s -> 649.375s (-0.09%)
Artifact size: 310.97 MiB -> 310.95 MiB (-0.01%)

klensy · 2024-02-23T08:49:39Z

Compiletest only accepts //@ and issues an error if an old-style directive is detected.

Don't see any error now when running COMPILETEST_FORCE_STAGE0=1 python x.py --stage=0 t tests/run-make/issue-85441 --force-rerun

compiletest directives in tests/run-make Makefiles currently not migrated, is it intentionally left for other pr?

jieyouxu · 2024-02-23T09:28:35Z

Compiletest only accepts //@ and issues an error if an old-style directive is detected.

Don't see any error now when running COMPILETEST_FORCE_STAGE0=1 python x.py --stage=0 t tests/run-make/issue-85441 --force-rerun

compiletest directives in tests/run-make Makefiles currently not migrated, is it intentionally left for other pr?

The compiletest directives in Makefiles that use # are not migrated, because... the plan is rip the Makefiles out in favor of plain rust files, i.e. #113026. That PR is still on-going.